Recursive Algorithms for Distributed Forests of Octrees

The forest-of-octrees approach to parallel adaptive mesh refinement and coarsening (AMR) has recently been demonstrated in the context of a number of large-scale PDE-based applications. Although linear octrees, which store only leaf octants, have an underlying tree structure by definition, it is not often exploited in previously published mesh-related algorithms. This is because the branches are not explicitly stored, and because the topological relationships in meshes, such as the adjacency between cells, introduce dependencies that do not respect the octree hierarchy. In this work we combine hierarchical and topological relationships between octree branches to design efficient recursive algorithms. We present three important algorithms with recursive implementations. The first is a parallel search for leaves matching any of a set of multiple search criteria. The second is a ghost layer construction algorithm that handles arbitrarily refined octrees that are not covered by previous algorithms, which require a 2:1 condition between neighboring leaves. The third is a universal mesh topology iterator. This iterator visits every cell in a domain partition, as well as every interface (face, edge and corner) between these cells. The iterator calculates the local topological information for every interface that it visits, taking into account the nonconforming interfaces that increase the complexity of describing the local topology. To demonstrate the utility of the topology iterator, we use it to compute the numbering and encoding of higher-order $C^0$ nodal basis functions. We analyze the complexity of the new recursive algorithms theoretically, and assess their performance, both in terms of single-processor efficiency and in terms of parallel scalability, demonstrating good weak and strong scaling up to 458k cores of the JUQUEEN supercomputer.Comment: 35 pages, 15 figures, 3 table

Localisation of gamma-ray interaction points in thick monolithic CeBr3 and LaBr3:Ce scintillators

Localisation of gamma-ray interaction points in monolithic scintillator crystals can simplify the design and improve the performance of a future Compton telescope for gamma-ray astronomy. In this paper we compare the position resolution of three monolithic scintillators: a 28x28x20 mm3 (length x breadth x thickness) LaBr3:Ce crystal, a 25x25x20 mm3 CeBr3 crystal and a 25x25x10 mm3 CeBr3 crystal. Each crystal was encapsulated and coupled to an array of 4x4 silicon photomultipliers through an optical window. The measurements were conducted using 81 keV and 356 keV gamma-rays from a collimated 133Ba source. The 3D position reconstruction of interaction points was performed using artificial neural networks trained with experimental data. Although the position resolution was significantly better for the thinner crystal, the 20 mm thick CeBr3 crystal showed an acceptable resolution of about 5.4 mm FWHM for the x and y coordinates, and 7.8 mm FWHM for the z-coordinate (crystal depth) at 356 keV. These values were obtained from the full position scans of the crystal sides. The position resolution of the LaBr3:Ce crystal was found to be considerably worse, presumably due to the highly diffusive optical in- terface between the crystal and the optical window of the enclosure. The energy resolution (FWHM) measured for 662 keV gamma-rays was 4.0% for LaBr3:Ce and 5.5% for CeBr3. The same crystals equipped with a PMT (Hamamatsu R6322-100) gave an energy resolution of 3.0% and 4.7%, respectively

An extreme-scale implicit solver for complex PDEs: highly heterogeneous flow in earth's mantle

Mantle convection is the fundamental physical process within earth's interior responsible for the thermal and geological evolution of the planet, including plate tectonics. The mantle is modeled as a viscous, incompressible, non-Newtonian fluid. The wide range of spatial scales, extreme variability and anisotropy in material properties, and severely nonlinear rheology have made global mantle convection modeling with realistic parameters prohibitive. Here we present a new implicit solver that exhibits optimal algorithmic performance and is capable of extreme scaling for hard PDE problems, such as mantle convection. To maximize accuracy and minimize runtime, the solver incorporates a number of advances, including aggressive multi-octree adaptivity, mixed continuous-discontinuous discretization, arbitrarily-high-order accuracy, hybrid spectral/geometric/algebraic multigrid, and novel Schur-complement preconditioning. These features present enormous challenges for extreme scalability. We demonstrate that---contrary to conventional wisdom---algorithmically optimal implicit solvers can be designed that scale out to 1.5 million cores for severely nonlinear, ill-conditioned, heterogeneous, and anisotropic PDEs